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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the 
Office  of  the  Under  Secretary  of  Defense  (Personnel  and  Readiness)  under  a  task  entitled 
“Improved  Methodologies  for  Relating  Flying  Hour  Activity  to  Operational  Readiness 
and  Safety  Measures.”  The  objective  of  this  portion  of  the  task  was  to  facilitate 
development  of  quantitative  relationships  between  the  capability  of  aviation  units  to 
perform  their  assigned  missions  and  the  levels  of  training  resources  available  to  them. 

This  work  was  reviewed  within  IDA  by  Bruce  N.  Angier,  Jesse  Orlansky,  and 
John  C.  Tillson. 
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EXECUTIVE  SUMMARY 


The  flying  hour  program  is  the  key  element  in  air  crew  continuation  training,  but 
there  is  considerable  uncertainty  about  the  value  of  maintaining  flying  hours  at  the  current 
level.  Flying  hours  are  an  attractive  target  for  congressional  budget  reductions  because 
money  can  be  saved  immediately.  With  lower  international  tensions  and  the 
accompanying  reduction  of  military  forces,  there  will  be  a  tendency  to  reduce  flying  hour 
programs.  In  this  environment,  it  is  important  to  have  a  better  idea  of  just  what  the 
implications  of  possible  reductions  in  flying  hours  are  for  the  performance  of  United 
States  military  air  crews  and  for  our  ability  to  effectively  mobilize  aviation  assets. 

This  report  develops  quantitative  relationships  between  how  much  air  crews  fly 
and  how  well  they  perform  important  aspects  of  their  missions.  The  current  study  is  the 
latest  in  a  series  of  statistical  analyses  that  examines  both  the  long-term  and  short-term 
effects  of  flying  hours  on  performance.  The  previous  studies  examined  four  aviation 
missions  or  skill  areas: 

•  the  quality  of  F-14  and  A-7  aircraft  landings  aboard  aircraft  carriers; 

•  the  accuracy  with  which  Marine  aviators  dropped  bombs  from  AV-8B, 
F/A- 1 8,  and  F-4S  aircraft; 

•  the  performance  of  F-14  fighters  during  opposed  air-combat-maneuvering 
exercises  on  an  instrumented  range; 

•  Marine  Corps  bombing  using  a  larger  database;  and 

•  Air  Force  C-130  tactical  airdrops. 

The  last  two  investigations  also  examined  the  contribution  of  simulator  training  to 
air  crew  performance.  Simulator  training  was  found  to  make  a  significant,  cost-effective 
contribution  to  performance.  The  findings  confirmed  the  existence  of  both  short-  and 
long-term  positive  effects  of  flying  hours  on  air  crew  performance.  In  addition,  the 
relative  magnitude  of  these  effects  was  estimated. 

This  study  examines  the  performance  of  Navy  Patrol  air  crews  and  extends  the 
previous  work  in  important  ways.  The  data  were  derived  from  graded  torpedo  attack 
exercises  (Torpexes)  flown  on  an  instrumented  range.  The  nature  of  the  Torpex,  a 
competitive  exercise  required  for  crew  qualification,  is  such  that  it  tests  the  crew 
members  in  several  aspects  of  the  maritime  patrol  mission.  The  aircraft  enters  the 
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exercise  area  with  the  target  submerged.  The  crew  must  detect,  classify,  track,  and 
successfully  launch  an  exercise  torpedo  in  order  to  qualify  in  the  exercise.  In  addition,  the 
P-3  used  for  the  patrol  mission  is  a  multiperson  aircraft,  and  it  was  possible  to  isolate  the 
training  effects  on  the  total  mission  performance  of  different  crew  members. 

The  crew  members  of  primary  interest  to  this  analysis  comprise  the  Tactical 
Nucleus  (TACNUC),  which  consists  of  two  officers,  the  aircraft  commander  (AC)  and 
the  tactical  coordinator  (TACCO),  and  two  enlisted  members,  the  first  acoustic  operator 
(SI)  and  the  electronic  warfare  operator  (S3).  The  aircraft  commander  is  a  Naval  Aviator 
responsible  for  flight  operations  and  safety  of  flight.  If  designated  as  the  mission 
commander,  he  is  also  responsible  for  the  conduct  of  the  overall  tactical  mission.  The 
TACCO  is  a  Naval  Flight  Officer  (NFO)  responsible  for  keeping  track  of  the  total  tactical 
picture  and  coordinating  the  work  of  the  other  members  of  the  crew.  His  station  is  the 
nerve  center  of  the  aircraft,  where  information  is  displayed,  analyzed  and  interpreted. 
Either  the  AC  or  the  TACCO  may  be  designated  Mission  Commander.  The  acoustic 
operator  uses  acoustic  processing  equipment  that  displays  and  assists  in  the  interpretation 
of  sonobuoy  information  to  detect,  classify,  and  localize  targets.  The  electronic  warfare 
operator  uses  radar  or  electronic  countermeasures  (ECM)  equipment  for  early  target 
detection  and  Magnetic  Anomaly  Detection  (MAD)  equipment  for  final  target 
localization. 

DATA 

Performance  data  were  obtained  from  Patrol  Wing  Five  at  Brunswick  Naval  Air 
Station,  Maine.  Flying  hour  and  simulator  training  data  were  obtained  from  daily  flight 
records  and  pilot  and  NFO  career  flying-hour  summaries  maintained  by  the  Sea  Logistics 
Center  at  Mechanicsburg,  Pennsylvania.  The  database  includes  mission  performance  data 
and  flight  and  simulator  experience  data  for  pilots,  TACCOs,  acoustic  operators,  and 
electronic  operators  from  five  operational  squadrons  flying  P-3  aircraft  on  torpedo  attack 
exercises.  Career  and  daily  flying  hours  were  available  for  pilots  and  TACCOs.  Only 
daily  flight  and  simulator  records  were  available  for  enlisted  crew  members.  Performance 
data  were  obtained  for  56  Torpex  missions.  Twenty-one  percent  of  these  were  flown 
against  an  actual  submarine,  and  the  remainder  were  flown  against  a  MK  30  target 
simulator. 
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ANALYSIS 


The  central  hypothesis  of  this  analysis  is  that  training  improves  performance. 
Thus,  crew  members  with  more  career  flight  and  simulator  hours  are  expected  to  detect, 
localize  and  track  the  target  with  greater  proficiency,  and  place  their  torpedoes  more 
accurately.  We  also  hypothesize  that,  holding  career  training  experience  constant, 
increases  in  recent  flying  and  simulator  experience  are  associated  with  more  accurate 
attacks. 

These  hypotheses  were  tested  using  a  Cobb-Douglas  production  function 
formulation.  To  gain  additional  insight  into  the  effect  of  experience  on  Torpex  scores,  we 
also  formulated  the  model  in  the  form  of  a  generalized  translog  production  function.  We 
hypothesize  that  interactions  exist  (1)  between  flying  hours  and  simulator  hours  and 
(2)  between  long-term  and  short-term  variables.  The  latter  formulation  turned  out  to 
provide  the  better  estimates. 

FINDINGS  AND  CONCLUSIONS 

The  findings  of  the  current  study  confirm  the  existence  of  both  short-  and  long¬ 
term  positive  effects  of  flying  hours  and  short-term  effects  of  simulator  hours  on  air  crew 
performance.  The  results  are  consistent  with  previous  studies,  in  that  the  long-term  effect 
is  stronger  than  the  short-term  effect.  We  were  unable  to  examine  long-term  flying  or 
simulator  effects  for  the  enlisted  crew  members  because  of  data  limitations.  We  were  able 
to  estimate  the  effect  on  performance  of  air  crew  simulator  hours  for  the  previous  30 
days. 

Table  S-1  summarizes  the  impact  on  performance  of  a  10%  decrease  in  flying 
hours  from  the  then-eurrent  levels. 


Table  S-1.  Decrease  in  Performance  (Torpex  Score)  for  10%  Decrease  in 
Three  Measures  of  Flyin^Simulator  Experience 


Career 

Recent 

Recent 

Flying 

Flying 

Simulator 

Hours 

Hours 

Hours 

Pilot  Aircraft  Commander 

0.4% 

_ 

TACCO  Mission  CDR 

0.2% 

_ 

First  Acoustics  Operator 

— 

0.04% 

0.2%a 

Total 

a  ^1.  rr^  *  * 

0.6% 

0.04% 

0.2% 

Because  the  TACCO,  First  Acoustics  Operator  and  Electronics  Operator  fly  and  exercise 
in  the  simulator  as  a  crew  the  short-term  effects  of  these  three  could  not  be  estimated 
together.  We  interpret  the  results  for  the  First  Acoustics  Operator  as  the  crew  effect. 
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Existing  data  on  performance  can  be  used  to  develop  relationships  between 
air  crew  performance  and  both  long-  and  short-term  experience  variables  that 
reflect  the  impact  of  variations  in  the  flying  hour  program. 

Maritime  patrol  Torpex  scores  are  influenced  by  the  number  of  career  aircraft 
commander  hours  of  the  pilot,  mission  commander  hours  of  the  TACCO,  and 
recent  flying  hours  of  the  acoustic  operator. 

This  finding  that  long-term  effects  are  quantitatively  more  important  than 
short-term  effects  implies  that  it  may  be  unwise  to  reduce  overall  air  crew 
experience  with  the  thought  that,  when  needed,  crews  can  be  brought  to  full 
effectiveness  with  a  short  period  of  intensive  training. 

A  10%  reduction  in  flying  hours,  both  career  and  recent,  is  estimated  to 
decrease  Torpex  scores  by  a  little  more  than  0.6%. 

On  the  margin,  the  partial  effect  of  recent  simulator  hours — that  is,  the  effect 
on  proficiency  of  one  simulator  hour  more  or  less — is  greater  than  that  of 
short-term  flying  hours.  (The  table  shows  that  a  10%  change  in  recent  flying 
has  a  greater  impact  than  a  10%  change  in  recent  simulator  use,  but  10% 
amounts  to  many  more  flight  hours  than  simulator  hours).  This  indicates  that 
the  additional  use  of  simulators  would  be  cost  effective. 

This  analysis  is  important  because  it  demonstrates  our  ability  to  estimate  the 
effect  of  flying  and  simulator  hours  on  performance  over  nearly  the  full  range 
of  the  maritime  patrol  mission — detection,  classification,  tracking,  and  attack. 

Because  of  multicollinearity,  we  were  unable  to  disentangle  the  effects  of  all 
of  the  variables.  In  particular,  we  were  unable  to  completely  disentangle  the 
individual  effects  of  each  member  of  the  TACNUC.  We  conjecture  that  this 
stems  from  the  fact  that  the  TACCO,  SI,  and  S3  fly  and  train  in  the  simulator 
together.  We  were  therefore  able  to  sort  out  the  short-term  variables  in  terms 
of  the  total  crew. 

We  cannot  predict  the  total  long-term  effect  of  additional  flying  because  of 
the  absence  of  enlisted  career  flying  hour  data. 

The  data  used  in  this  study  reflect  operations  before  the  collapse  of  the  Soviet 
Union,  when  maritime  patrol  squadrons  averaged  approximately  45  flying 
hours  per  month  per  crew  over  the  entire  1 8-month  training  cycle.  This  has 
been  reduced  to  as  little  as  30  hours  per  month.  We  would  expect  the 
quantitative  findings  to  be  different  for  current  data. 

It  would  appear  to  be  worthwhile  to  analyze  the  data  generated  under  the 
current  flying  hour  levels.  It  is  too  great  a  leap  to  extrapolate  our  findings  to  a 
33%  decrease  in  flying  hours  (45  to  30  hours  per  month  per  crew).  An 
experiment  using  current  data  could  greatly  increase  our  understanding  of  the 
effect  of  marginal  changes  in  career  flying  hours  and  both  short-term  flying 


and  simulator  hours  on  Torpex  scores.  A  more  complete  data  set  would  be 
available  provided  the  assistance  of  the  wing  and  squadrons  was  enlisted  for 
a  period  of  approximately  1  year.  With  more  complete  records,  including 
enlisted  career  flying  hours  and  a  complete  record  of  crews  for  every 
competitive  exercise,  many  of  the  problems  with  the  current  data  could  be 
eliminated.  Other  data  problems,  such  as  the  lack  of  career  TACCO  and 
enlisted  simulator  hours,  would  require  a  major  change  in  Navy  record 
keeping  procedures  over  a  longer  period  of  time. 

RECOMMENDATIONS 

We  recommend  that  controlled  experiments  be  conducted  and  a  more  complete 
database  constructed  to  perform  the  following: 

•  studies  designed  to  gain  a  better  understanding  of  the  cost-effectiveness 
implications  of  changes  in  flying  hour  budgets  for  different  missions,  aircraft 
types,  and  crew  positions. 

•  analyses  aimed  at  further  documenting  the  marginal  effects  of  changes  in 
simulator  and  actual  flying  hours  on  performance  at  current  and  projected 
levels. 

To  better  understand  the  full  potential  costs  of  reduced  air  crew  training,  the 
results  of  studies  such  as  this  one  should  be  incorporated  into  combat  analyses. 

Our  final  recommendation  is  to  consider  the  following  policy  actions: 

•  Moving  to  a  more  senior  mix  of  personnel.  This  could  involve  encouraging 
longer  careers,  perhaps  by  modifying  retirement  policies  and  incentives. 

•  Keeping  individuals  with  hard-to-train  operational  skills,  such  as  pilots,  in 
operational  billets  as  much  as  possible,  consistent  with  the  need  for  trained 
and  educated  leaders. 

•  Making  use  of  experienced  personnel  through  maximum  practical  reliance  on 
the  reserve  components  (particularly  in  aviation)  should  be  encouraged. 
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I.  INTRODUCTION 


The  flying  hour  program  is  the  key  element  in  air  crew  continuation  training,  but 
there  is  considerable  uncertainty  about  the  value  of  maintaining  flying  hours  at  the  current 
level.  Flying  hours  are  an  attractive  target  for  congressional  budget  reductions  because 
money  can  be  saved  immediately.  Over  $10  billion  a  year  are  spent  on  the  flying  hour 
programs  of  all  services.  With  lower  international  tensions  and  the  accompanying 
reduction  of  military  forces,  there  will  be  a  tendency  to  reduce  flying  hour  programs.  In 
this  environment,  it  is  important  to  have  a  better  idea  of  just  what  the  implications  of 
possible  reductions  in  flying  hours  are  for  the  performance  of  United  States  military  air 
crews  and  for  our  ability  to  effectively  mobilize  aviation  assets. 

This  paper  develops  quantitative  relationships  between  how  much  air  crews  fly 
and  how  well  they  perform  important  aspects  of  their  missions.  The  current  study  is  an 
extension  of  statistical  analyses  reported  in  references  [1]  and  [2].  The  referenced 
analyses  examined  both  the  long-term  and  short-term  effects  of  flying  hours  on 
performance.  Five  empirical  investigations  examined  four  aviation  missions  or  skill 
areas: 

•  the  quality  of  F-14  and  A-7  aircraft  landings  aboard  aircraft  carriers, 

•  the  accuracy  with  which  Marine  aviators  dropped  bombs  from  AV-8B, 
F/A-18,  and  F-4S  aircraft, 

•  the  performance  of  F-14  fighters  during  opposed  air-combat-maneuvering 
exercises  on  an  instrumented  range, 

•  Marine  Corps  bombing  using  a  larger  database,  and 

•  US AF  C- 130  tactical  airdrops. 

Reference  [2]  also  examined  the  contribution  of  simulator  training  to  air  crew 
performance.  Simulator  training  was  found  to  make  a  significant,  cost-effective 
contribution  to  performance.  The  findings  confirmed  the  existence  of  both  short-  and 
long-term  positive  effects  of  flying  hours  on  air  crew  performance.  In  addition,  the 
relative  magnitude  of  these  effects  were  estimated.  Table  I-l  summarizes  the  impact  on 
performance  of  a  10%  decrease  in  flying  hours  from  the  then  current  levels,  based  on  the 
analyses  in  references  [1]  and  [2].  All  the  effects  shown  in  the  table  are  statistically 
significant. 
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Table  1-1.  Decrease  in  Performance  for  10%  Cuts  in 
Four  Measures  of  Flying/Simulator  Experience 


Performance  Measure 

Career 

Flying 

Hours 

Recent 

Flying 

Hours 

Total 

Flying 

Hours 

Career 

Simulator 

Hours 

Unsatisfactory  Landings^ 

6.9% 

2.6% 

9.5% 

— 

Air-to  Air  Combat^’^ 

Probability  Red  Kills  Blue 

6.3% 

2.9% 

9.2% 

— 

Probability  Blue  Kills  Red 

-2.6% 

-2.2% 

-4.8% 

— 

Marine  Bombing  Miss  Distance^ 

1.2% 

0.6% 

1.8% 

0.3% 

C-130  Tactical  Airdrop  Miss  Distance^ 

Copilot 

0.5% 

— 

0.5% 

1.0% 

Navigator 

— 

2.3% 

2.3% 

— 

Total 

0.5% 

2.3% 

2.8% 

1.0% 

3  Source:  Reference  [1]. 

^  Red  are  aggressor  aircraft;  Blue  are  friendly  aircraft.  The  flying  hour  decrease  is  only  for  Blue  air  crews, 
c  Source:  Reference  [2]. 


This  study  reports  the  results  of  additional  statistical  analyses  of  both  the  long- 
and  short-term  effects  of  flying  hours  and  simulator  hours  on  performance.  This  analysis 
examines  the  performance  of  Navy  Patrol  air  crews  and  extends  the  previous  work  in 
important  ways.  The  data  were  derived  from  graded  torpedo  attack  exercises  (Torpexes) 
flown  on  an  instrumented  range.  The  Torpex,  a  competitive  exercise  which  is  required  for 
crew  qualification,  tests  the  crew  members  in  several  aspects  of  the  maritime  patrol 
mission.  The  aircraft  enters  the  exercise  area  with  the  target  submerged.  To  qualify  in  the 
exercise,  the  crew  must  detect,  classify,  track,  and  successfully  launch  an  exercise 
torpedo.  The  P-3  used  for  the  patrol  mission  is  a  multiperson  aircraft;  it  was  possible  to 
isolate  the  training  effects  on  total  mission  performance  of  different  crew  members. 

Our  findings  confirm  the  existence  of  both  short-  and  long-term  positive  effects  of 
flying  hours  and  short-term  effects  of  simulator  hours  on  air  crew  performance.  Because 
of  data  limitations,  we  were  unable  to  examine  the  long-term  flying  or  simulator  effects 
for  the  enlisted  crew  members.  We  were  able  to  estimate  the  effect  on  performance  of  air 
crew  simulator  hours  for  the  previous  30  days. 

A  description  of  the  data,  the  analyses,  and  findings  are  presented  in  Chapter  n. 
Conclusions  and  recommendations  are  stated  in  Chapter  in. 
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II.  ANALYSIS  OF  NAVY  PATROL  AIRCRAFT 
TORPEDO  ATTACK  EXERCISES 

This  chapter  reports  the  results  of  an  investigation  of  the  relation  between  P-3  air 
crew  training,  measured  by  flying  and  simulator  hours,  and  performance  in  detecting, 
classifying,  tracking,  and  attacking  submarines  and  simulated  targets.  This  analysis  treats 
a  multi-person  mission,  and  we  were  able  to  estimate  the  contribution  of  short-term 
simulator  hours  to  mission  performance,  which  had  not  been  possible  in  our  earlier  work. 
When  mission  performance  depends  on  more  than  one  person,  it  is  generally  more 
difficult  to  isolate  the  effect  of  experience  of  each  crew  member  on  crew  performance. 
For  this  analysis,  we  were  able  to  separate  the  training  effects  of  the  pilot  from  those  of 
the  tactical  coordinator  and  acoustic  equipment  operator.  Further,  the  performance  data 
capture  the  output  of  the  training  process  far  more  completely  than  some  of  the  measures 
that  have  been  used  in  the  past  (Marine  air-to-ground  miss  distance,  for  example). 

A.  DESCRIPTION  OF  THE  AIRCRAFT  AND  CREW 

The  Navy  Maritime  Patrol  aircraft  is  the  P-3  Orion,  a  long-range  land-based 
multi-engine  aircraft  that  is  manned  on  operational  missions  by  a  crew  of  approximately 
twelve — ^flve  officers  and  seven  enlisted  personnel.  Crews  are  organized  for  maximum 
stability  and  once  assigned  to  a  crew,  personnel  generally  remain  with  that  crew  for  the 
duration  of  their  squadron  tour.  The  training  cycle  is  approximately  18  months,  with  the 
squadron  being  deployed  during  the  last  6  months  of  the  cycle.  Crews  are  formed  in  the 
early  part  of  this  cycle  and  fly  together  during  the  work-up  period  and  subsequent 
deployment. 

The  crew  members  of  primary  interest  to  our  analysis  comprise  the  Tactical 
Nucleus  (TACNUC),  which  consists  of  two  officers,  the  aircraft  commander  (AC)  and 
the  tactical  coordinator  (TACCO),  and  two  enlisted  members,  the  first  acoustic  operator 
(SI)  and  the  electronic  warfare  operator  (S3).  The  aircraft  commander  is  a  Naval  Aviator 
responsible  for  flight  operations  and  safety  of  flight.  The  TACCO  is  a  Naval  Flight 
Officer  (NFO)  responsible  for  keeping  track  of  the  total  tactical  picture  and  coordinating 
the  work  of  the  other  members  of  the  crew.  His  station  is  the  nerve  center  of  the  aircraft, 
where  information  is  displayed,  analyzed,  and  interpreted.  Either  the  AC  or  the  TACCO 
may  be  designated  Mission  Commander  with  overall  responsibility  for  the  mission.  The 
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acoustic  operator  uses  acoustic  processing  equipment  that  displays  and  assists  in  the 
interpretation  of  sonobuoy  information  to  detect,  classify,  and  localize  targets.  The 
electronic  warfare  operator  uses  radar  or  electronic  countermeasures  (ECM)  equipment 
for  early  target  detection  and  Magnetic  Anomaly  Detection  (MAD)  equipment  for  final 
target  localization. 

Precise  coordination  is  required  among  all  members  of  the  crew;  the  members  of 
the  TACNUC  are  key  to  the  tactical  operation  and  can  limit  the  capability  of  the  overall 
crew.  Members  of  the  TACNUC  must  be  aboard  and  manning  their  assigned  crew 
positions  when  certain  critical  events,  called  Competitive  Exercises  (COMPEXs),  are 
flown.  These  exercises  are  graded  by  the  wing  staff  and  contribute  to  the  crew  and 
squadron  readiness  rating.  When  one  member  of  the  TACNUC  is  transferred,  all  other 
crew  members  lose  their  COMPEX  qualifications  and  must  re-qualify  with  the 
transferee’s  replacement.  For  each  member  of  the  TACNUC,  there  is  at  least  one  backup 
crew  member  who  is  in  training  for  his  position.  The  copilot  is  in  training  for  AC,  the 
navigator — an  NFO — is  in  training  as  TACCO,  and  the  second  acoustics  operator  (S2)  is 
in  training  for  the  SI  position.  The  S2  may  also  be  in  training  for  the  S3  position,  and 
each  crew  may  have  other  enlisted  trainees  assigned  who  are  in  training  for  the  SI,  S2,  or 
S3,  positions.  If  a  member  of  the  TACNUC  is  due  to  be  rotated  during  the  training  cycle, 
his  backup  generally  qualifies  in  the  TACNUC  position  for  all  exercises  which  require 
qualification  as  a  crew.  The  backup  crew  member  can  then  be  designated  as  part  of  the 
TACNUC  at  the  appropriate  time  without  loss  of  crew  readiness  points. 

B.  DATA 

Data  for  this  study  were  obtained  from  Patrol  Wing  Five  at  Brunswick  Naval  Air 
Station,  Maine.  The  database  includes  mission  performance  data  and  experience  data  for 
pilots,  TACCOs,  acoustic  operators,  and  electronic  operators  from  five  operational 
squadrons  flying  P-3  aircraft  on  torpedo  attack  exercises.  The  missions  were  Torpexes, 
flown  for  TACNUC  qualification,  on  an  instrumented  range  located  in  either  the 
Roosevelt  Roads,  Puerto  Rico,  or  Key  West  operating  areas. 

The  grading  of  torpedo  attacks  is  more  complicated  than  for  the  bomb  or  rocket 
launches  analyzed  in  previous  studies.  For  those  analyses,  the  miss  distance  was  used  as 
the  measure  of  performance.  For  Torpexes,  the  course  and  speed  of  the  target  as  well  as 
the  placement  of  the  torpedo  and  aircraft  heading  must  be  taken  into  account.  The 
probability  of  a  hit  (Ph)  is  calculated  by  range  personnel,  and  depends  on  the  distance 
between  the  torpedo’s  point  of  entry  into  the  water  and  the  target  as  well  as  the  angle 
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between  the  entry  point  and  the  target  course.  The  Ph  also  depends  on  how  closely  the 
crew  estimates  the  target  speed.  The  drop  score  is  calculated  from  the  probability  of  hit.' 

Crew  qualification  requires  satisfactory  performance  in: 

•  Weapons  management — ^Weapon  properly  loaded,  and  all  settings  correct  and 
properly  reported  in  the  firing  report. 

•  Localization/Tracking — ^Localize  target  and  track  for  a  minimum  of  30 
minutes  prior  to  initial  attack. 

•  Attack  criteria: 

-  Target  tracked  to  attack  criteria  at  time  of  each  attack 

-  All  torpedoes  launched  within  weapon  parameters 

•  Attack  effectiveness: 

The  crew  attains  a  minimum  score  of  13.  Points  are  calculated  as 
follows: 

For  the  first  attack  the  number  of  points  equals  10  times  the  probability 
of  hit  calculated  by  range  personnel.  For  each  subsequent  attack  up  to  a 
total  of  four,  the  number  of  points  equals  five  times  the  probability  of 
hit.  A  minimum  of  two  and  a  maximum  of  four  attacks  may  therefore  be 
flown  to  attain  qualification,  and  the  maximum  number  of  points 
attainable  is  25. 

-  In  the  absence  of  the  above  scoring  criteria,  range  Operations  confirms 
that  the  torpedo  acquired  the  target. 

For  this  study  all  participants  flew  four  attacks.  Therefore,  the  number  of  points  is 
a  measure  of  the  sum  of  the  probability  of  hit  for  all  attacks.  Flying  hour  and  simulator 
training  data  were  obtained  from  daily  flight  records  and  from  pilot  and  NFO  career 
flying-hour  summaries  maintained  by  the  Sea  Logistics  Center  at  Mechanicsburg, 
Pennsylvania.  The  data  base  is  maintained  as  part  of  the  Naval  Flight  Information 
Reporting  System  (NAVFLIRS).  NAVFLIRS  is  an  automated  computer  data  base  which 
includes  the  flight-hour  history  of  all  pilots  and  NFOs  by  year,  aircraft  type  and  model, 
and  individual  flight  records  for  all  air  crew  personnel  by  squadron  and  sortie.  Pilot  daily 
flight  and  career  data  include  pilot  and  aircraft  commander  hours,  night  hours,  instrument 
hours,  and  simulator  hours.  Daily  and  career  flight  hours,  mission  commander  hours,  and 
night  hours  are  recorded  for  NFOs.  Simulator  hours  are  maintained  on  the  daily 


*  Only  30%  of  the  torpedoes  were  functional  exercise  torpedoes  (EXTORPS).  The  remainder  were 
shapes,  called  REXTORPS.  (REXTORPs  have  the  same  air-drop  characteristics  as  an  actual  torpedo, 
but  do  not  run  after  water  entry).  However,  the  type  of  torpedo  used  should  not  affect  the  score. 
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NAVFLIRS  records  for  NFOs,  but  are  not  transferred  to  the  career  files.  Only  daily  flight 
and  simulator  records  are  maintained  in  the  NAVFLIRS  data  base  for  enlisted  crew 
members.  Career  flight-hour  totals  are  recorded  in  each  enlisted  crew  member’s  log  book, 
but  these  stay  with  the  individual  so  were  not  available  for  crewmen  who  were  no  longer 
in  the  squadron  when  the  data  were  collected. 

Flight  and  simulator  histories  for  the  previous  7,  30,  and  60  days  were  constructed 
from  the  daily  flight  records.  Although  daily  flight  data  are  recorded  by  individual  by 
aircraft,  it  is  not  possible  to  tell  who  is  actually  occupying  each  crew  station  from  the 
daily  flight  record.  This  was  determined  from  available  monthly  crew  lists  published  by 
each  squadron. 

Performance  data  were  obtained  for  56  Torpex  missions.  Twenty-one  percent  of 
these  were  flown  against  an  actual  submarine,  and  the  remainder  against  a  MK  30  target 
simulator.  A  target  simulator,  which  looks  like  a  small  torpedo,  electronically  generates 
acoustic  and  magnetic  signatures  similar  to  those  of  a  submarine.  Target  simulators  are 
programmed  to  follow  a  pre-set  track  and  may  be  capable  of  reacting  to  active  acoustic 
detection  equipment,  depending  on  the  model. 

Table  II-l  is  a  summary  of  the  data  used  in  the  analysis.  Scores  were  calculated  by 
range  personnel.  The  score  is  related  to  the  probability  that  a  war  shot  torpedo  would 
cause  lethal  damage,  given  a  specific  entry  location  and  identical  conditions  (i.e.,  aircraft 
track;  submarine  course,  speed,  and  depth;  and  acoustic  conditions).  Flight-hour 
information  was  calculated  from  NAVFLIRS  data. 

C.  ANALYSIS 
1.  Hypotheses 

The  central  hypothesis  of  this  analysis  is  that  training  improves  performance. 
Thus,  crew  members  with  more  career  flight  and  simulator  hours  are  expected  to  detect, 
locate  and  track  the  target  with  greater  proficiency,  and  place  their  torpedoes  more 
accurately.  We  also  hypothesize  that  more  recent  flying  and  simulator  experience,  holding 
career  training  experience  constant,  is  associated  with  more  accurate  attacks. 

An  additional  hypothesis  of  interest  is  that  crew  stability  enhances  performance. 
However,  because  of  the  lack  of  complete  data  on  when  each  member  joined  the  crew,  we 
were  unable  to  test  this  hypothesis. 
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Table  11-1.  Data  Used  in  the  Analysis  of  Navy  P-3  Torpex  Missions 


Variable 

Minimum 

Mean 

Maximum 

Score 

9.9 

19.4 

23.9 

Pilot  Aircraft  Commander  Hours 

26.0 

747.6 

3267.5 

TACCO  Mission  Commander  Hours 

0.0 

287.4 

836.9 

Pilot  Flight  Hours  Previous  7  Days 

0.0 

6.5 

24.1 

Pilot  Flight  Hours  Previous  30  Days 

3.9 

26.5 

80.6 

Pilot  Flight  Hours  Previous  60  Days 

3.9 

54.7 

140.5 

Pilot  Career  Simulator  Hours 

29.4 

111.6 

292.0 

Pilot  Simulator  Hours  Previous  7  Days 

0.0 

1.0 

6.0 

Pilot  Simulator  Hours  Previous  30  Days 

0.0 

3.1 

17.0 

Pilot  Simulator  Hours  Previous  60  Days 

0.0 

5.3 

25.2 

TACCO  Flight  Hours  Previous  7  Days 

0.0 

7.0 

27.9 

TACCO  Flight  Hours  Previous  30  Days 

0.0 

27.6 

92.2 

TACCO  Flight  Hours  Previous  60  Days 

0.0 

56.1 

172.0 

TACCO  Simulator  Hours  Previous  7  Days 

0.0 

1.3 

6.6 

TACCO  Simulator  Hours  Previous  30  Days 

0.0 

4.0 

17.1 

TACCO  Simulator  Hours  Previous  60  Days 

0.0 

6.8 

23.3 

SI  Flight  Hours  Previous  7  Days 

0.0 

7.8 

37.6 

SI  Flight  Hours  Previous  30  Days 

0.0 

20.2 

131.0 

S 1  Right  Hours  Previous  60  Days 

0.0 

44.4 

182.3 

SI  Simulator  Hours  Previous  7  Days 

0.0 

1.3 

6.6 

S 1  Simulator  Hours  Previous  30  Days 

0.0 

4.0 

17.0 

SI  Simulator  Hours  Previous  60  Days 

0.0 

6.9 

24.0 

S3  Right  Hours  Previous  7  Days 

0.0 

7.4 

29.8 

S3  Right  Hours  Previous  30  Days 

0.0 

32.4 

121.7 

S3  Right  Hours  Previous  60  Days 

0.0 

61.2 

182.7 

S3  Simulator  Hours  Previous  7  Days 

0.0 

1.2 

6.6 

S3  Simulator  Hours  Previous  30  Days 

0.0 

3.5 

16.2 

S3  Simulator  Hours  Previous  60  Days 

0.0 

5.8 

22.3 

Percentage  Actual  SSN  Targets 

21.0 

2.  The  Model 

a.  The  Cobb-Douglas  Production  Function  Model 

We  postulate  a  log-log  relationship  based  on  the  form  of  the  raw  data,  our  a  priori 
beliefs  about  learning,  and  previous  analyses  of  similar  data.  We  expect  learning  to 
exhibit  diminishing  returns  to  increases  in  the  experience  variables.  The  estimating 
equation  takes  the  following  form: 

LnScore  =  Lnbo  4-  bi  x  LnHp,  +  b2  x  LnHpst  +  ba  x  LnHn  +  b4  x  LnHpz  +  bs  x  LnHpsz  (1) 
+  ba  X  LnH,z  +  b?  x  LnHtsz  +  bs  x  LnHsiz  +  b9  x  LnHsisz  +  bio  x  LnHsSz 
-I-  bii  X  LnHsSsz  +  bi2  x  SSN 
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where: 


Ln 

Score 

H 


pt 


Hps, 

Ht, 
H, 


pz 


H, 


psz 


H,z 

Htsz 

Hslz 

Hslsz 

Hs3z 

Hs3sz 

SSN 


bo  through  bi2 


the  natural  log 

Torpex  score  assigned  by  range  personnel 
pilot  career  flying  hours 
pilot  career  simulator  hours 
TACCO  career  flying  hours 

pilot  flying  hours  in  the  previous  z  days,  where  z  =  7, 30,  or  60  days 

pilot  simulator  hours  in  the  previous  z  days,  where  z  =  7,  30,  or  60 
days 

TACCO  flying  hours  in  the  previous  z  days 
TACCO  simulator  hours  in  the  previous  z  days 
S 1  flying  hours  in  the  previous  z  days 
SI  simulator  hours  in  the  previous  z  days 
S3  flying  hours  in  the  previous  z  days 
S3  simulator  hours  in  the  previous  z  days 

a  dummy  variable  taking  the  value  one  for  an  attack  against  an  actual 
SSN  (nuclear  attack  submarine)  and  0  otherwise 

coefficients  to  be  estimated 


The  coefficients  bi  and  b3  measure  the  effect  on  performance  of  additional  pilot 
and  TACCO  career  flying  hours,  respectively,  holding  career  simulator  hours  constant. 
Both  bi  and  b3  are  expected  to  be  positive — more  experience  should  be  reflected  in  higher 
scores.  The  coefficient  hj  measures  the  effect  on  performance  of  additional  pilot  career 
simulator  hours,  holding  career  flying  hours  constant.  We  conjecture  that  scores  will 
increase  if  simulator  hours  increase  while  holding  flying  hours  constant.  We  therefore 
expect  ba  to  be  positive. 


The  coefficients  b4,  be,  bg  and  bio  measure  the  effect  on  the  score  of  pilot, 
TACCO,  SI,  and  S3  flying  hours,  respectively,  in  the  previous  7,  30  or  60  days.  We 
expect  these  coefficients  to  be  positive.  The  coefficients  bs,  by,  b9  and  bn  measure  the 
effect  on  the  score  of  pilot,  TACCO,  SI,  and  S3  simulator  time,  respectively,  in  the 
previous  7,  30  or  60  days.  We  expect  these  coefficients  to  be  positive  as  well.  We  cannot 
use  two  different  short-term  variables  for  the  same  crew  position  in  the  same  equation 
because  these  variables  are  highly  correlated.  We  might  be  able  to  use  different  short¬ 
term  variables  for  different  crew  positions,  since  it  is  not  clear  that  recently  honed  skills 
depreciate  at  the  same  rate  for  different  crew  positions. 
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SSN  is  a  control  variable.  Missions  flown  against  an  actual  SSN  are  considered  to 
be  more  difficult  than  those  flown  against  a  simulated  target.  An  SSN  can  take  calculated 
evasive  action  throughout  the  exercise.  We  therefore  expect  bi2  to  be  negative. 

b.  The  Model  With  Interactions 

In  order  to  gain  additional  insight  into  the  effect  of  experience  on  Torpex  scores, 
we  also  formulated  the  model  in  the  form  of  a  generalized  translog  production  function. 
We  hypothesize  that  interactions  exist  between  flying  hours  and  simulator  hours,  and 
between  long-term  and  short-term  variables.  The  estimating  equation  then  takes  the  form 
of  equation  (2).  The  terms  in  braces  are  the  interaction  terms.  We  have  shown  only  3 
interaction  terms  for  the  sake  of  brevity,  since  there  are  as  many  possible  combinations  as 
there  are  pairs  of  terms  in  equation  (1). 

LnScore  =  Lnbo  -i-  bi  x  LnHp,  -i-  b2  x  LnHps,  -h  ba  x  LnHtt  +  b4  x  LnHp^  +  bs  x  LnHpsz  (2) 
+  bg  X  LnHtz  -t-  b7  X  LnHtsz  +  bg  x  LnHsiz  +  b9  x  LnHsisz  +  bio  x  LnHsSz 
-I-  b|  I  X  LnHsSsz  +  {bi,2  x  LnHp,  x  LnHps,  +  b6,7  x  LnH,z  x  LnH,sz 
ba^o  X  LnH,,  x  LnH,z  + ... }  -t-  bi2  x  SSN , 

where  the  definitions  of  terms  are  the  same  as  for  equation  (1)  and  the  bij  are  the 
coefficients  of  the  interaction  terms. 

We  expect  the  coefficients  of  the  individual  terms  to  be  the  same  as  for  equation 
(1).  The  coefficients  of  the  interaction  terms  are  expected  to  be  negative.  The  signs  of  the 
interaction  coefficients  correspond  to  the  signs  of  the  cross  partial  derivatives  of  score 
with  respect  to  each  variable  in  the  interaction  term.  For  example,  we  hypothesize  that  for 
higher  levels  of  aircraft  commander  career  flying  hours,  the  marginal  increase  in  score 
with  increasing  career  simulator  hours  would  be  smaller,  since  more  highly  trained  people 
are  expected  to  gain  less  from  additional  training. 

3.  Methodology 

Equations  (1)  and  (2)  were  estimated  using  ordinary  least  squares.  Because  many 
of  the  minimum  values  of  the  independent  variables  are  zero,  a  small  constant  was  added 
to  all  independent  variable  observations.  This  is  a  commonly  used  Econometric  technique 
and  prevents  the  natural  log  from  being  indeterminate  (minus  infinity). 
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D.  FINDINGS 


1.  MulticoIIinearity 

The  effect  of  changes  in  the  pilot  career  simulator  variable  and  the  effect  of 
changes  in  pilot  career  flying  hours  could  not  be  estimated  together  because  they  are 
collinear.  Pilots  with  more  career  flying  hours  tend  to  have  more  simulator  hours  as  well. 
This  means  that  the  estimated  effect  of  more  flying  hours  includes  the  effect  of  additional 
simulator  hours  that  usually  go  along  with  them.  In  addition,  pilots  with  more  career 
flying  hours  tend  to  have  fewer  recent  simulator  hours.  We  are  unable  to  separate  out  the 
individual  effects  of  these  three  variables.  Perhaps  this  is  not  a  serious  problem.  Although 
simulator  events  include  coordinated  tactical  exercises,  pilots  concentrate  on  instruments, 
emergency  procedures,  and  basic  flying  skills.  Pilot  simulator  time  is  therefore  not  as 
directly  related  to  torpedo  attacks  as  is  the  simulator  time  of  the  remainder  of  the  crew. 
Although  we  did  find  a  significant  statistical  relationship  between  pilot  simulator  hours 
and  Torpex  scores,  pilot  career  flight  and  simulator  hours  could  not  be  estimated  together. 
For  estimated  equations  where  career  simulator  hours  are  significant,  the  simulator 
variable  acts  (to  some  extent)  as  a  proxy  for  flight  hours. 

In  the  case  of  the  remaining  TACNUC  members,  it  was  even  more  difficult  to 
untangle  the  effect  of  flight  and  simulator  hours  on  score.  We  had  no  long-term 
experience  variables  which  might  be  useful  in  distinguishing  the  individual  partial  effects 
of  the  enlisted  crew  members.  In  addition,  the  TACCO,  SI,  and  S3  participate  together  in 
nearly  all  simulator  and  flight  events.  The  short-term  flight  and  simulator  variables  are 
therefore  highly  correlated  across  individuals.  In  order  to  try  to  draw  some  meaningful 
conclusions,  we  searched  for  a  variable  which  would  serve  as  a  proxy  for  the  short-term 
experience  of  the  TACCO  and  enlisted  sensor  operators  as  a  team.2 

2.  Variables  Used  in  the  Estimating  Equations 

The  pilot  and  TACCO  career  experience  variables  used  in  the  estimating 
equations  are  Aircraft  Commander  and  Mission  Commander  hours,  respectively.  The 
short-term  variables  used  in  the  equations  are  S I  flying  hours  in  the  previous  60  days  and 
S 1  simulator  hours  in  the  previous  30  days.  Recent  flying  hours  for  the  pilot  and  TACCO 
were  not  found  to  be  significant,  and  are  omitted  from  the  final  equations.  The  S3 


^  See  any  standard  Econometrics  text  for  a  discussion  of  the  effects  and  possible  solutions  to  the 
multicollinearity  problem.  The  direction  we  have  taken  is  to  drop  some  of  the  variables  and  accept  the 
remaining  variables  as  proxies  for  those  not  included  in  the  equation. 
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variables  could  not  be  estimated  in  the  same  equation  with  the  SI  variables,  and  the  SI 
effects  were  much  more  significant  than  the  S3  effects  alone.  This  is  not  surprising 
because  the  SI  plays  a  more  important  role  in  this  particular  exercise.  None  of  the 
interactions  between  flight  hour  and  simulator  variables  were  statistically  significant. 
However,  the  interaction  between  TACCO  mission  commander  hours  and  SI  flying  hours 
in  the  previous  60  days  could  be  estimated.  The  estimated  coefficients  of  equations  (1) 
and  (2)  using  this  set  of  variables  are  reported  in  Table  II-2. 


Table  il-2.  Determinants  of  P-3  Torpex  Performance 


Dependent  Variable:  Ln  Score 

Independent  Variable 

Coefficient 

Equation  ( 1 )  Equation  (2) 

Intercept 

2.60 

(16.2)*** 

2.59 

(16.5)*** 

Ln  of  total  pilot  aircraft  commander  hours  (Hpt). 

0.0385 

(1.68)* 

0.0364 

(1.65) 

Ln  TACCO  total  mission  commander  hours(Htt) 

0.0168 

(1.49) 

0.0325 

(2.39)** 

Ln  S-1  flight  hours  previous  60  days  (Hsif6o) 

0.0224 

(1.98)** 

0.0420 

(2.82)*** 

Ln  S-1  simulator  hours  previous  30  days  (Hsisso) 

TACCO  career  mission  commander-S- 1  60-day  flight 
hour  interaction  (Ln  Hu  x  Ln  Hsif6o) 

0.0186 

(2.241)** 

0.0170 

(2.10)** 

-0.0067 

(1.95)* 

Submarine  target  dummy  variable 

-0.0733 

(1.14) 

-0.0797 

(1.28) 

Number  of  Observations 

53 

53 

Adjusted  R-square 

Equation  (2)  Hu  elasticity 

Equation  (2)  Hsif60  elasticity 

.27 

.31 

0.0231 

0.0039 

Note:  Numbers  in  parentheses  are  t-statistics. 


*  Significant  at  the  0. 1 0  level. 
**  Significant  at  the  0.05  level. 
***  Significant  at  the  0.01  level. 


3.  Elasticities  of  Score  With  Respect  to  Variables  Used  in  the  Equations 

The  results  shown  in  Table  11-2  are  consistent  with  all  of  the  hypotheses  discussed 
above.  All  of  the  coefficients  have  the  expected  sign.  All  of  the  coefficients  in  equation 
(1)  except  for  TACCO  career  mission  commander  hours  and  the  SSN  dummy  are 
significant  at  the  0.10  level  of  significance  or  better  using  a  two-tailed  test.  The  TACCO 
career  mission  commander  hours  coefficient  is  significant  at  the  0.14  level.  The  SSN 
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dummy  is  a  control  variable,  and  our  primary  interest  is  that  the  sign  be  as  expected.  All 
of  the  coefficients  in  equation  (2)  except  for  Pilot  Career  Aircraft  Commander  hours  and 
the  SSN  dummy  are  significant  at  the  0.10  level  or  better  using  a  two-tailed  test.  The  Pilot 
Career  Aircraft  Commander  hours  coefficient  is  significant  at  the  0.1081  level. 

Equations  (1)  and  (2)  explain  respectively  only  0.27  and  0.31  of  the  variation  in 
Ln  Score.  However,  the  R^s  are  above  the  norm  relative  to  other  studies  which  analyzed 
this  type  of  detailed  cross-sectional  flying  hour  data.  These  R^s  are  comparable  to  those 
obtained  for  our  Marine  Corps  bombing  analysis  [2].  This  percentage  of  explained 
variation  means  that  the  equations  cannot  precisely  predict  a  given  Torpex  score  based  on 
the  explanatory  variables  in  these  equations.  The  goal  of  this  paper  is  not,  however,  to 
predict  individual  scores.  Rather,  it  is  to  estimate  the  effect  of  variations  in  training 
experience  on  the  average  scores  of  a  population  on  a  large  number  of  exercises.  The 
statistical  significance  of  the  flying  and  simulator  hour  coefficients  indicates  that  it  is 
adequate  for  this  purpose  and,  therefore,  for  setting  policy  on  flying  and  simulator  hours. 

Because  the  equations  were  estimated  in  log-log  form  the  coefficients  are 
estimated  elasticities.  Elasticity  is  the  percentage  change  in  the  crew’s  score  with  a  1% 
change  in  the  independent  variable.  For  the  variables  that  appear  in  the  interaction  term, 
the  elasticity  must  be  calculated  using  the  coefficient  of  the  interaction  term  and  the 
average  values  of  the  independent  variables,  as  well  as  the  coefficients  of  the  terms 
involving  the  individual  variables. 

Table  11-3  shows  the  estimated  elasticities  and  partial  effects  of  the  experience 
variables  for  equation  (2)  calculated  at  the  mean.  The  partial  effect  is  the  change  in  the 
dependent  variable  with  a  unit  change  in  the  independent  variable.  For  example,  if  SI 
flight  hours  in  the  previous  60  days  were  to  change  by  1  hour  we  would  expect  Score  to 
change  by  0.002. 


Table  11-3.  Elasticities  and  Partial  Effects  of  Torpex  Scores 
With  Respect  to  the  Experience  Variables  (Equation  2) 


Indepencjent  Variable 

Elasticity 

Partial  Effect 

Pilot  Career  Aircraft  CommancJer  Hours  (Hpt) 

0.0364 

TACCO  Career  Mission  CommancJer  Hours  (Htt) 

0.0231 

SI  Flight  Hours  in  Previous  60  Days  (Hsih6o) 

0.0039 

SI  Simulator  Hours  in  Previous  30  Days  (Hsis3o) 

0.0170 
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4.  Graphical  Representation  of  the  Results 

The  effects  reported  in  Table  II-2  are  shown  graphically  over  the  full  range  of  the 
data  in  Figures  II-l  through  11-4.3  These  graphs  depict  the  effect  of  changing  each 
experience  variable  while  holding  all  other  variables  constant.  In  all  cases  the 
independent  variables  which  are  not  being  examined  are  held  at  their  observed  means.^ 
All  of  the  graphs  show  that  the  marginal  improvement  for  an  additional  flying  or 
simulator  hour  is  greater  for  crewmen  with  less  experience.  This  is  the  expected  result 
and  is  a  characteristic  of  the  functional  form.  Functional  forms  that  did  not  exhibit  this 
characteristic  did  not  fit  the  data  as  well. 


Pilot  Aircraft  Commander  Hours 

Figure  11-1.  P-3  Torpedo  Attack  Score  Versus 
Pilot  Career  Aircraft  Commander  Hours 


In  Figures  II- 1  through  II-4,  the  first,  third,  and  fifth  values  are  the  minimum,  mean,  and  maximum 
observed  values,  respectively.  The  second  and  fourth  points  are  the  mid-points  between  those  values.  In 
cases  where  the  minimum  value  is  zero,  a  small  constant  was  added  to  prevent  the  predicted  value  of 
the  dependent  variable  from  being  indeterminate  (minus  infinity). 

In  order  to  graph  the  estimated  equation  we  made  the  transformation: 

Score  =  exp{Ln  Score}  =  explSj  biXj). 

However,  this  is  a  biased  estimate  of  Score  because  the  mean  of  the  transformed  residual,  which  is  non¬ 
zero,  is  omitted.  For  Ln  Uj  distributed  N(0,s2),  Uj  is  distributed  log-normal  with  mean  exp{0+(s2/2)}. 
We  therefore  include  the  correction  exp{s2/2}  where  s  is  the  estimated  standard  deviation  (root  mean 
square  error).  The  transformed  equation  is: 

Score  =  exp{Ln  Score)  =  exp{I|  bjXj}  expisVl], 
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Figure  11-2.  P-3  Torpedo  Attack  Score  Versus 
TACCO  Career  Mission  Commander  Hours 


Figure  11-3.  P-3  Torpedo  Attack  Score  Versus 
S-1  Simulator  Hours  in  the  Previous  30  Days 
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S-1  Flight  Hours  Previous  60  Days 

Figure  11-4.  P-3  Torpedo  Attack  Score  Versus 
S-1  Flying  Hours  in  the  Previous  60  Days 


E.  SUMMARY 

The  findings  indicate  that  flying  and  simulator  hours  in  the  short-term  and  flying 
hours  in  the  long-term  have  a  significant  effect  on  Torpex  performance.  Career  flying 
hours  for  the  aircraft  commander  and  TACCO  have  greater  percentage  effects  on  Torpex 
scores  than  the  short-term  variables.  This  has  been  a  consistent  finding  in  past  studies. 
Because  of  data  limitations,  we  were  unable  to  estimate  the  long-term  effects  for  the 
enlisted  crew  members. 

Because  of  the  nature  of  the  Torpex  exercise,  the  findings  are  also  applicable  to  a 
greater  part  of  the  full  mission  than  previous  studies,  other  than  our  analysis  of  air-to-air 
combat  exercises.  For  exarnple,  our  analysis  of  Marine  bombing,  reference  [2],  examined 
only  the  bombing  run  and  delivery,  and  did  not  address  success  in  reaching  or  egressing 
from  the  target  area.  Torpex  scores  reflect  the  crew’s  success  in  detecting,  classifying  and 
tracking  the  target,  and  the  accuracy  of  the  weapon  delivery. 

On  the  margin,  the  partial  effect  of  recent  simulator  hours  is  greater  than  for  the 
short-term  flying  hour  variables.  Since  simulator  hours  are  less  expensive  than  flying 
hours,  this  implies  that  simulator  hours  are  cost-effective.  This  effect  may  apply  only  to 
small  changes  near  the  observed  mix  of  flying  and  simulator  hours.  The  effect  of  larger 
increases  in  simulator  hours,  especially  when  combined  with  decreases  in  flying  hours, 
cannot  be  discerned  from  our  analysis. 
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Because  of  multicollinearity,  we  were  unable  to  disentangle  the  effects  of  all  of 
the  variables;  however,  we  were  able  to  sort  out  the  short-term  variables  in  terms  of  the 
total  crew.  In  other  words,  the  significant  results  for  the  SI  short-term  flying  hour  and 
simulator  variables  may  proxy  for  the  TACCO,  SI,  and  S3  as  a  crew.  We,  of  course, 
cannot  state  this  as  a  fact  because  the  S 1  is  a  key  crew  member  throughout  the  detection, 
classification,  tracking,  and  attack  phases  of  the  exercise.  The  effect  we  observed  could 
be  attributable  to  the  S 1  alone.^ 

The  estimation  problem  is  partly  a  result  of  the  way  in  which  crews  train,  but  to  a 
great  extent  this  is  a  data  collection  problem.  The  VP  crews  fly  many  graded  competitive 
exercises  during  each  training  cycle.  The  scores  for  these  exercises  are  recorded  along 
with  the  squadron  and  crew  number.  However,  the  records  needed  to  identify  the 
individual  members  of  the  crew  are  kept  only  for  approximately  9  months,  and  those  that 
are  retained  are  sporadic  for  periods  earlier  than  the  most  recent  2  or  3  months.  Torpex 
exercise  data  were  selected  for  this  analysis  because  we  were  able  to  reconstruct  the 
Torpex  crews  from  flight  records  and  a  scattering  of  crew  lists.  The  crews  fly  to  a  Naval 
Air  Station  near  the  range  prior  to  each  exercise,  and  with  this  information  we  could 
identify  the  aircraft  and  crew  members.  However,  we  could  identify  the  crew  positions  for 
only  about  a  year’s  worth  of  data.  This  greatly  limited  the  number  of  available 
observations.  In  general,  graded  exercise  score  data  do  not  identify  the  aircraft  or  the 
individual  crew  members,  so  it  was  impossible  to  identify  the  crew  members  from 
NAVFLIRS  data  for  exercises  other  than  Torpexes.  Additional  data  could  have  another 
beneficial  effect,  since  some  competitive  exercises,  other  than  Torpedoes,  are  flown  while 
squadrons  are  deployed.  Simulators  are  not  available  to  deployed  squadrons,  so  these 
exercises  could  produce  smaller  correlations  between  the  short-term  flying  and  simulator 
hour  variables.  A  second  serious  data  problem  is  the  lack  of  a  complete  record  of  enlisted 
career  flying  hours. 

Based  on  these  results,  we  can  make  some  quantitative  statements  with  respect  to 
short-term  experience  for  the  crew  and  long-term  experience  for  the  aircraft  commander 
and  TACCO  crew  positions.  Based  on  the  estimated  coefficients  for  equation  (2),  if  the 
number  of  flying  hours  were  reduced  by  10%  for  a  short  period  of  time  the  average 
Torpex  score  would  decrease  by  0.04%.  A  similar  decrease  in  crew  simulator  hours 


^  We  also  experimented  with  linear  combinations  of  the  short-term  variables  for  these  crew  positions 
with  mixed  results.  Linear  combinations  of  the  TACCO  and  SI  short-term  simulator  and  flying  hour 
variables  were  highly  significant,  but  the  coefficients  could  not  be  estimated  accurately  with  the  pilot 
and  TACCO  career  hours. 
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would  result  in  a  decrease  of  approximately  0.2%.  If  the  decrease  in  flying  hours 
continued  until  pilot  and  TACCO  career  experience  degraded  by  10%,  then  Torpex  scores 
would  decrease  an  additional  0.6%,  the  sum  of  the  AC  and  TACCO  coefficients 
multiplied  by  10.  We  cannot  predict  the  total  long-term  effect  because  of  the  absence  of 
enlisted  career  flying  hour  data.  Table  II-4  summarizes  these  results. 


Table  11-4.  Decrease  in  Performance  (Torpex  Score)  for  10%  Decrease 
in  Three  Measures  of  Fiying/Simulator  Experience 


Career 

Recent 

Recent 

Flying 

Flying 

Simulator 

Hours 

Hours 

Hours 

Pilot  Aircraft  Commander 

0.4% 

_ 

_ 

TACCO  Mission  CDR 

0.2% 

_ 

_ 

First  Acoustics  Operator 

— 

0.04% 

0.2%^ 

Total 

0.6% 

0.04% 

0.2% 

Because  the  TACCO,  First  Acoustics  Operator  and  Electronics  Operator  fly  and  exercise 
in  the  simulator  as  a  crew  the  short-term  effects  of  these  three  could  not  be  estimated 
together.  We  interpret  the  results  for  the  First  Acoustics  Operator  as  the  crew  effect. 


The  data  used  in  this  study  reflect  operations  before  the  collapse  of  the  Soviet 
Union,  when  maritime  patrol  squadrons  averaged  approximately  45  flying  hours  per 
month  per  crew  over  the  entire  18-month  training  cycle.  This  has  been  reduced  to  as  little 
as  30  hours  per  month.  We  would  expect  the  quantitative  findings  to  be  different  for 
current  data. 

The  above  effects  are  difficult  to  interpret  because  we  do  not  know  the  effect  of  a 
percentage  decrease  in  average  Torpex  scores.  Although  the  range  of  scores  in  our  study 
was  14,  the  minimum  score  was  3  standard  deviations  below  the  mean  of  19.4,  and  most 
of  the  observations  were  bunched  near  the  mean.  A  decrease  in  the  mean  score  of  1% 
might  represent  more  or  less  than  a  1%  decrease  in  submarine  kills. 
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III.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Existing  data  on  performance  can  be  used  to  develop  relationships  between  air 
crew  performance  and  both  long-  and  short-term  experience  variables  that  reflect  the 
impact  of  variations  in  the  flying  hour  program.  This  analysis  is  important  because  it 
demonstrates  our  ability  to  estimate  the  effect  of  flying  and  simulator  hours  on 
performance  over  nearly  the  full  range  of  the  maritime  patrol  mission — detection, 
classification,  tracking,  and  attack. 

The  following  conclusions  can  be  drawn  from  this  analysis. 

•  Maritime  patrol  Torpex  scores  are  influenced  by  the  number  of  career  aircraft 
commander  hours  of  the  pilot,  mission  commander  hours  of  the  TACCO,  and 
recent  flying  hours  of  the  acoustic  operator. 

•  Recent  acoustic  operator  simulator  hours  influence  Torpex  scores 
significantly,  and  simulator  hours  appear  to  be  cost-effective. 

•  The  long-term  effects  are  quantitatively  more  important  than  short-term 
effects.  This  is  consistent  with  previous  analyses  (see  references  [1]  and  [2]) 
and  has  important  policy  implications  during  a  period  of  shrinking  force 
structure  and  budgets.  This  finding  implies  that  it  may  be  unwise  to  reduce 
overall  air  crew  experience  with  the  thought  that,  when  needed,  crews  can  be 
brought  to  full  effectiveness  with  a  short  period  of  intensive  training. 

•  A  10%  reduction  in  flying  hours,  both  career  and  recent,  is  estimated  to 
decrease  Torpex  scores  by  a  little  more  than  0.6%. 

•  Although  we  were  unable  to  completely  disentangle  the  individual  effects  of 
each  member  of  the  TACNUC,  we  were  able  to  separate  the  long-term  effects 
of  the  pilot  and  TACCO  from  the  short-term  effects  of  the  TACCO  and 
enlisted  crew  members.  With  a  more  complete  data  set  we  believe  that  more 
meaningful  results,  particularly  the  long-term  effects  of  the  training  given 
enlisted  crew  members,  are  obtainable. 

•  It  would  appear  to  be  worthwhile  to  analyze  the  data  generated  under  the 
current  flying  hour  levels.  It  is  too  great  a  leap  to  extrapolate  our  findings  to  a 
33%  decrease  in  flying  hours  (45  to  30  hours  per  month  per  crew).  An 
experiment  using  current  data  could  greatly  increase  our  understanding  of  the 
effect  of  marginal  changes  in  career  flying  hours  and  both  short-term  flying 
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and  simulator  hours  on  Torpex  scores.  A  more  complete  data  set  would  be 
available  provided  the  assistance  of  the  wing  and  squadrons  was  enlisted  for 
a  period  of  approximately  one  year.  With  more  complete  records,  including 
enlisted  career  flying  hours  and  a  complete  record  of  crews  for  every 
competitive  exercise,  many  of  the  problems  with  the  current  data  could  be 
eliminated.  In  particular,  if  we  had  a  complete  record  of  when  each  member 
joined  the  crew,  we  could  test  the  hypothesis  that  crew  stability  enhances 
performance.  Other  data  problems,  such  as  the  lack  of  career  TACCO  and 
enlisted  simulator  hours,  would  require  a  major  change  in  Navy  record 
keeping  procedures  over  a  longer  period  of  time. 

This  study  is  highly  complementary  to  the  work  reported  in  references  [1]  and  [2] 
because  it  treats  a  multi-person  aircraft  and  because  it  treats  nearly  the  entire  mission  of 
the  aircraft.  Once  again  an  analysis  of  objective  data  has  shown  that  reductions  in  flying 
hours  will  lead  to  degradations  in  performance. 

The  findings  are  consistent  with  earlier  work,  but  they  also  indicate  that  the 
database  should  be  updated  and  expanded  for  the  following  reasons: 

•  The  findings  could  be  strengthened  by  an  increase  in  both  the  number  of 
observations  and  the  variety  of  exercise  types. 

•  The  lack  of  career  flying  hours  for  the  enlisted  crew  members  seriously  limits 
the  completeness  of  the  analysis. 

•  Recent  decreases  in  average  monthly  crew  flying  hours  could  dramatically 
change  the  implications  of  these  findings  with  regard  to  the  effect  of  marginal 
changes  in  both  flying  hours  and  simulator  hours. 

The  kinds  of  relationships  developed  in  this  paper  do  not,  by  themselves,  answer 
the  increasingly  urgent  questions  posed  by  recent  and  on-going  defense  budget 
reductions.  Neither  we  nor  other  researchers  have  adequately  addressed  the  cost- 
effectiveness  of  flying  hour  reductions.  Savings  in  flying-hour  budgets  may  result  in 
higher  costs  in  other  areas  in  peacetime  or  they  may  have  adverse  affects  in  wartime. 

For  example,  the  peacetime  cost  of  reduced  training  should  include  the  planes  and 
pilots  that  might  be  lost  because  of  increased  accident  rates.  A  recurring  observation 
made  by  air  crewmen  throughout  this  series  of  studies  is  that  they  must  fly  with  some 
minimum  frequency,  which  they  perceive  as  necessary  for  safety,  or  they  will  not  remain 
in  the  service.  Reduced  air  crew  retention  increases  personnel  acquisition  and  training 
costs. 
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Wartime  costs  involve  the  possibility  of  less  favorable  combat  outcomes  because 
of  reduced  pilot  proficiency.  Analyses  based  on  the  use  of  constructive  simulations  of 
warfare  could  give  greater  insight  into  the  possible  magnitude  of  these  costs.  * 

The  importance  of  aggregate  career  training  experience  indicates  that  a  more 
experienced  force  is  a  readier  force.  It  seems  clear  that  extended  tours  in  staff  and 
management  billets  decrease  average  career  flight  experience.  Further  analysis  is  needed 
to  address  the  implications  of  the  results  of  this  (and  related)  studies  for  some  key  force 
and  career  management  policy  issues,  such  as: 

•  What  the  average  career  experience  level  should  be;  how  much  training  is 
enough? 

•  How  to  maintain  the  proper  balance  of  operational  skills,  officer  education 
and  leadership  training. 

•  How  to  better  use  the  reserve  component — which  has  more  experienced 
pilots  to  attain  the  best  mix  of  operational  and  administrative  skills. 

B.  RECOMMENDATIONS 

We  recommend  that  controlled  experiments  be  conducted  and  a  more  complete 
database  constructed  to  perform  the  following: 

•  Studies  designed  to  gain  a  better  understanding  of  the  cost-effectiveness 
implications  of  changes  in  flying  hour  budgets  for  different  missions,  aircraft 
types,  and  crew  positions. 

•  Analyses  aimed  at  further  documenting  the  marginal  effects  of  changes  in 
simulator  and  actual  flying  hours  on  performance  at  current  and  projected 
levels. 

To  better  understand  the  full  potential  costs  of  reduced  air  crew  training,  the 
results  of  studies  such  as  this  one  should  be  incorporated  into  combat  analyses. 

Our  final  recommendation  is  to  consider  the  following  policy  actions: 

•  Moving  to  a  more  senior  mix  of  personnel.  This  could  involve  encouraging 
longer  careers,  perhaps  by  modifying  retirement  policies  and  incentives. 

•  Keeping  individuals  with  hard-to-train  operational  skills,  such  as  pilots,  in 
operational  billets  as  much  as  possible,  consistent  with  the  need  for  trained 
and  educated  leaders. 


*  Some  preliminary  work  of  this  type  has  been  performed  and  is  available  in  briefing  form  from  the 
authors  under  the  title,  “The  Cost-Effectiveness  of  Flying  Hours  and  Simulation.” 
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Making  use  of  experienced  personnel  through  maximum  practical  reliance  on 
the  reserve  components  (particularly  in  aviation). 


m-4 


REFERENCES 


REFERENCES 


[1]  Hammon,  Colin  P.,  and  Stanley  A.  Horowitz.  “Flying  Hours  and  Aircrew 
Performance.”  Institute  for  Defense  Analyses,  Paper  P-2379,  March  1990. 

[2]  Hammon,  Colin  P.,  and  Stanley  A.  Horowitz.  “Relating  Flying  Hours  to  Aircrew 
Performance:  Evidence  for  Attack  and  Transport  Missions.”  Institute  for  Defense 
Analyses,  Paper  P-2609,  June  1992. 


A-1 


ABBREVIATIONS 


ABBREVIATIONS 


AC 

Aircraft  Commander 

COMPEX 

Competitive  Exercise 

ECM 

electronic  countermeasures 

EXTORP 

exercise  torpedo 

MAD 

Magnetic  Anomaly  Detection 

NAVFLIRS 

Naval  Flight  Information  Reporting  System 

NFO 

Naval  Flight  Officer 

REXTORP 

torpedo  shape 

SI 

first  acoustic  operator 

S2 

second  acoustic  operator 

S3 

electronic  warfare  operator 

SSN 

nuclear  attack  submarine 

TACCO 

Tactical  Coordinator 

TACNUC 

Tactical  Nucleus 

Torpex 

Torpedo  Exercise 

VP 

designation  for  patrol  aircraft 
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